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^ABSTRACT 

Based on seven years of surveying students, parents, 
and educators, the following conclusions were reached: (1) 
preliminary telephoning of graduates to verify addresses did not 
produce higher returns from those contacted; (2) cover letters 
printed on more expensive colored-letterhead stationary did not 
produce a higher return rate than the same letter photocopied all in 
black; (3) no differences .were found in return rates from graduates 
whose surveys were mailed to arrive on Wednesday and those whose 
questionnaires were to arrive on Saturday; (4) fewer responses were 
received from graduates who received "cute" reminders than from those 
whose reminders were more businesslike; (5) no di f f erence was found 
in return rate on parent questionnaires when half were mailed first 
class and half bulk rate; (6) on computer-printed teacher 
questionnaires, half of the teachers who returned questionnaires did 
not respond to the free-response items; and (7) free-response items 
where blank spaces were left for responses were completed more often 
by administrators than those where lines were printed in . Additional 
post hoc findings and an optical-scan form on which to print unique 
questionnaires are also described. (BW) 
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PRACTICE MAKES PERFECT? 
SKILLS GAINED IN SEVEN YEARS OF QUESTIONNAIRES 

Looking back at the title of this paper after a lapse of six months, I found 
that the question mark best expresses where we are now. Last summer, after 
seven years of surveying students, graduates, teachers, administrators, 
counselors, and parents, a paper on our progress seemed appropriate. 

.Not only had we learned a number of things about what matters and what doesn't, 
but we also had learned how to learn. The first "finding" came by accident: 
we tried to telephone all of a sample of graduates to verify their addresses, 
but only. reached about half. Tfiere was no difference in the return rate from 

■r ' 

those contacted and those not. We were surprised. 

After that we began to set up "experiments": take a random sample and assign 
one randomly divided half to one condition and the other half to a different 
condition. Even changing just one aspect of the procedure for any one survey 
you can draw a number of conclusions if you send out enough different question- 
naires over enough years, and keep good records. 

We tried to find ways to improve return rates, ways to save money, and ways to 
aet more information. We also tried to limit, the number of questionnaires 
staff received by collecting information for other District departments. The 
things we learned this way are summarized below, and in Attachment A are 
listed the. individual surveys we have conducted, with factors we've varied, am 
the return rates. 

• Preliminary telephoning of graduates to verify addresses did 
not produce higher returns .from those contacted. 

• Cover letters printed on (more expensive) colored-letterhead 
stationary did not produce a higher return rate than the same 
letter photocopied all in black, j 

• No differences' were found in return rates from graduates whose 
surveys were mailed to arrive on Wednesday and those whose 
questionnaires were to arrive on Saturday. (Reminders were 
mailed on the same schedule.) 

• Fewer responses were received irom graduates who received 
"cute" reminders than from those "whose reminders were more 
businesslike. 

•No difference was found in return rate on parent-question- \ 
naires when half were mailed first class arid half bulk rate. \ 
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On computer-printed teacher questionnaires, half of the 
teachers who returned questionnaires did not respond to the 
free-response items. 

• Free-response items where blank spaces were left for responses 
were completed more often by administrators than those where 
lines were printed in. 

We learned some other things post hoc, just by looking at what came back. 

• Reminders (with a new survey included) substantially 
increased returns. 

•The highest response rate to a parent questionnaire on 
counseling was from parents of elementary students, with 
the lowest response from parents of high school students p 
and the rate from parents of junior high students in the ' W 

middle. 

• Over- sampling of graduates with gr-ade averages below 85 
for, the- Former Student Questionnaire achieved a returned 
sample that matched the District on percentages in each 
grade range. 

• Return rates from both teachers and administrators have 
improved over three years as the annual survey became more 
routine and we gave feedback on summaries of results. 

• The lowest return rate ever from graduates came the year 
we mailed questionnaires in April instead of February. 

• We could limit the number of' items any one teacher or 
administrator questionnaire included yet answer a large 
pool of items by matrix sampling and computer generating 
a unique survey for each teacher and administrator. 

We were pleased with the process. We were using systematic experimental 
design to answer questions for our evaluations. We were providing more 
information for more staff with more confidence in our results. So why the 
big question mark? 

Although computer generating unique questionnaires for teachers and adminis 
trators was obviously the way to go - few items per person, true random 
assignment of items, specific items to targeted individuals 
or groups, and many items included for staff districtwide - processing was 
costly. Keypunching 1,500 unique questionnaires took time. We needed 
thing we had never seen. We wanted an optical-f.can form on which to print 
(by computer) our unique questionnaires. Attachment B shows what we have 
developed. 
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We ordered 250,000 of these last November^ Needless to say there were some 
bugs in the system the first time we used them, for 15,000 high school student 
questionnaires. But they did print, and they did scan, and nobody mutilated 
the form too badly. Th ? system is here to stay, ^ 

So far this year we have used this form for three sets of surveys, for students 
teachers, and administrators. All the District's high school students received 
questionnaires. Each student was asked a set of seven standard items about 
vocational courses and summer-job needs, and each jgtudent also received two 
randomly assigned. questions from a pool of 13 items. This gave as responses 
from 1,400 to 1,600 student per item. These were printed with teacher and 
student name, and arranged to be boxed by school. When they were returned, 
they .were scanned and the most urgently needed information was immediately 
ready for the schools in print outs. 

For the administrator and teacher surveys, quick scanning of returned forms 
let us print reminders and envelope labels 'only for personnel who had not yet 
returned theirs. We had a pool of- 120 items for teachers, but asked each, 
teacher to respond to only about 25. 

So this is an innovation -.hat will streamline our whole survey operation. 
There were some problems with these first efforts, but the procedure seemed 
understandable to those who had to use it. There were no wrinkled or torn 
forms, and only one staple got hung in the scanner. We may make a few changes 
in the form layout before we print more this summer. We will probably try 
the format with younger, students next year - junior high and possibly upper 
elementary. We may use it with next year's Former Student Questionnaire. And 
next. year we will start "experimenting" again: 

• Send half the graduates a printed form and half, the scannable 
form to check whether they respond better or worse to it. 

• Think, up some items to get at staff reactions to the forms 
themselves. 

• Use a teacher advisory group we. are forming to pilot new 
items and formats. 

All this has ranged rather far from "what any little school can do to get 
better return rates." We have, entered High Tech - we've traded low-skill 
tasks for automation, but this means we use an experienced data analyst 
and extensive computer files. We've come a long way from wondering whether 
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blue or yellow paper would produce better returns, and I don ! t think we are 
going back. I would still like to include quarters with half the parent 
questionnaires, but elon't think the finance office would approve purchase 
orders for rolls of quarters! 

Probably the most important thing we've learned is that we can get usable 
returns on surveys. We T ve moved -\£rom "send questions home with every student 
and report the 10% that come back M to random samples small enough to send 
multiple reminders. We use computers extensively, and highly trained personnel. 
We spent time on the items (that 1 3 another whole paper) both on gathering them 
from across the District and on writing them well. We try to learn from the 
mistakes we continue to make. We continue to plan the questionnaires and the 
surveying procedures and to try new ideas. 



I 
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Attachment A 
(Page 1 of 2) 



STAFF SURVEYS 



DATE 


uKUUr 


SAMPLE 


1/17/77 


Teachers 


100% HS = 954 t 


1/17/77 


Counselors 


100% HS = 49 t 


1/17/77 


Non-teaching 
Personnel 


100% HS = 174 : 


12/07/78 


Professionals 
(New 5 due 
evaluation) 


120 


5/02/79 


Professionals 


514 


3/04/81 


Teachers 


18% All= -578 


3/19/81 


Administrators 


50% All=. 156 


3/02/82 


Teachers 


100% of 1 HS 
* 100% of 1 JHS 
100% of Migrant 
50% of remaining 
= 1582 


3/01/82 


Administrators 


100% of those not 
surveyed '81 
50% of new 

=» 155 


2/16/83 


Teachers 


100% cf those not 
surveyed f 82- 
50% of those who • 
were 100% '32 
50% of new 

= 1614 


2/ 14/83 


Administrators 


10tf% * 315' 


2/2o/82 


Counselors 

i 


100% = 76 



FACTORS EFFECTING RETURNS 



RETURN RA;E 



15 items memo to principals 5 supervisors 
about their staff gettingthe 
questionnaires no reminder 



1 reminder 

25 items, half to each lialf of total group 
1 reminder 

17 items 1 reminder 

63 items (9 to 14 items per teacher) 
computer generated, matrix sampled 

some items to specific teachers or groups, others random 
1 reminder after 3 weeks 

7W0 OPEN-RESPONSE ITEMS — - ' 

» 

23 itess 1 reminder 

THREE OPEN-RESPONSE ITEMS: 

HALF THE SURVEYS WITH 3 LINES PRINTED 

HALF THE SURVEYS WITH BLANK SPACE FOR RESPONSE 

102 items (about 15 items per ^teacher) 
computer generated, matrix sampled 

some items to. specific teachers or groups, others random 



83% 
100% 
88% 

58% 

76% 

71% 

82% 
80% 



-53% 

85% 

-55% 
-69% 

84% 



52 items (from 10 to some central to 33 -for some elementary) 90% 
computer generated; matrix sampled 

some items to administrators with specific programs, the 

rest assigned by grade level j ^ j STRATORS 94% 

ELEMENTARY CAMPUS ——93% 

6 SECONDARY CAMPUS 79% 

28 items for elementary, 31 items for secondary ^ 100% 
items about counseling 

no reminder in mail, telephone follow up 



PARENT SU 
2/14/83 


RVEY 
Parents 


Parents of: 
6th graders=200 
8th graders=200 - 

11th graders=200 • 


14 items for secondary, 12 items for elementary 62% 
about counseling 

33 returned for incorrect address, ■ 
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GRADUATE SURVEYS <. 

EACH OF THESE SURVEYS WAS PRINTED FRONT AND BACK ON LIGHT SLUE PAPER, WITH ABOUT 35 ITEMS, ON CONTINUING < EDUCATION , 
EMPLOYMENT STATUS, SATISF\CTION WITH THEIR HIGH SCHOOL EDUCATION, AND GENERAL "ATTITUDES TOWARD THE DISTRICT. 

• Each rfent out with a computer-printed address label and the envelope* was metered with first-class postage, 

• Each contained a cover lettsr signed by the Superintendent of Schools, 

• Each included a first-class postage metered self -addressed return envelope. 

•All reminders included "reminder" cover letter, new questionnaire, and another metered return envelope, 

• Each questionnaire had a. sequence number stamped in the upper right comer, but not the graduate's name, 

• Each cover letter mentioned the number, and stressed that no graduate's name would be linked with responses. 



DATE 


CLASS OF 


NUMBER OF GRADUATES 


SAMPLE SIZE 


UNDELIVERABLE 


FACTORS AFFECTING RETURNS RETURN RATE 


2/01/77 


1976 


3229 


254 




Half addresses verified by phone 73% 
3 reminders, 3rd special delivery 


2/08/78 


1977 


3416 


4S0 


8% 


3 reminders, 3rd special delivery 67% 


2/01/79 


1973 


3656 


522 


-JO. 

>■ f' 


T 

rrenminary letter i*/ u/ /o jjo 
HALF ORANGE LETTERHEAD COVER LETTER, 
HALF BLACK COPY— —BOTH RETURNS-—-™ 35% 
2 reminders, HALF WITH "CUTE" NOTES 
HALF WITH BUSINESSLIKE 
(623 of returns got 
businesslike reminders) 


1/30/ SO 


1979 


3403 


535 




I attai< t j -5 t" Vi cl mim i t*V r* £ fl ic e rtf 78 fifi* 

Letter witn bUiiunui/ or ti«io j ui » u u u t» 
data - 4/79 - sent via counselors . 
3 reminders 


2/03/31 


1980 


3144. 


566j~ 


6% 


3 reminders 66% 

HALF MAILED FRIDAY — 63%. 

(Before reminders, Tues. = 30% 
Fri. = 33%) 


1/15/81 


1977 


3416 


450 


25% 


2 reminders 40% 
(RETURN RATE FOR THOSE DELIVERED— 54%) 
Sample was graduates already surveyed 
2/08/78, 9 months after graduation-- 
all in original sample sent surveys 


4/3/83 


1982 


3357 


653 


* 7% 


1 reminder 46% 

(sample returned nntched the District 
.grade distribution, and was close to 
j the District for sex and ethnic dist.) 
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Attachment B 



USE NO, 2 PENCIL ONLY 



AUSTIN INDEPENDENT SCHOOL DISTRICT 

OFFICE OF RESEARCH AND EVALUATION _ ^ 
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A copy of this report may be obtained 
from the address below. Pub. No. ,$ ? d v i/ 1 



Q TCE OF RESEARCH AND EVALUATION, AISD, 
ERjO GUADALUPE, BOX 79, AUSTIN, TX. 78752 
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